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control system for i 
a laboratory environment. 


of the design and ux^ksnentaftioo of a modular, high perfor ma nce. paraDd computer 
Topics of consideration include system architecture, operator interface, and control execution. In 
a telerobotics test control configuration is used to obtain measurements on conanunacations and 


control loop tuning for use in an effective full scale operational system design. The feasibility of the sel e ct ed architectural 
approach has been successfully demonstrated. The modularity of the software and hardware enables ease of tran sp ort for use in 
the operational system, a j«r part inn r*f thi* pyr devoted-to-tb e distributed pardoning of the control algorithms and the 

p e r for ma nce measurements acquired during control system i mp kn a en t ationy^ (/ t * a 


2. Introduction 

Telerobots are manipulators designed to be controlled both directly by a computer and remotely by a human. This combina- 
tion method, called supervisory control, allows the operator to apply his intelligence to the task without having to maintain 
continuous control. Computational requirements for robot control either limit the complexity of the control system, or require 
prohibitively large computers to obtain the required cycle rate. Remote operation and human control requirements create 
additional environmental constraints. The design of a hierarchical, parallel computer system is described. It provides the 
required speed within the prescribed design limits at a minimum weight and size. A prototype system was built md measure- 
ments were made to verify several performance issues. 

3. System Description 

The space-based tele robot for which this computer system was designed (Grumman [ 1]) hes several aspects that drive the 
design of the computer system. 

The first major design requirement is imposed by the robot itself (Fig. 1). There are three arms, each with seven joints and an 
end-effector, a torso, a vision sy stem, and tools and test equipment. All this equipment is under computer control. Each arm 
must operate under control of several different types of control algorithms to provide flexibility to enhance operator productiv- 
ity. Two arms must be capable of coordinated action. The manipulation requirements impose definite compilation requi r ements. 
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A second major design driver results from the control devices used 10 operate in the various control inodes. In order to 
maximize productivity, a variety of control inputs and information displays are needed. Input control devices include a replica 
master. 6DOF stick controllers, voice control, a lightpen. a keyboard, and a touchbezel. Output devices include graphics 
display s, monitors, and force reflection <Fig. 2). Each of these require some data processing. 



Fig. 2 Telerobot Control Station Testbed 


I he third rtuior design driver is that the robot is ; emote f rom the control center. This creates a requirement for a communica- 
tor? link \rs> communication link imposes restrictions and trade-offs on total data bandwidth. 

\ Tina! design driver ;s the set ot different control algorithms required for human control and robotic control. These impose 
requirements tor htferent data .md v *»ntrol paths Different control algorithms have different Ux>p rates K>r stability, wit ha 
•a ’lai a J computations must 'v performed 

4. System ** pec i fit at ion 

I he te:eroh«*t s vauircd To run three different control algorithms Ki lateral Force Reflection iBFR). Resolved Rate (RR>. 
tnd \cf.e <’ vrr \f C I ": : Boih HI R and RR must have human control. \F (' uses computer control * »nl> . BFR 

i is arce .>*n:i;:.*:ii^at.on requirements rial a complete set .»! se:is,.r s 4i ;nais must ne passed both trom the masters to the 
slaves, aru: me 'iaves v 'he masters performance :s ontuined with updates of about H/. \FC has a required 

up fate tor ts command \edor «t about H/ in order to maintain stability u hen the manipulator is m c^tact with elements 

- ‘I N jy t \ irt-nit’e'!* 

Distributed pr-xesMiii: was .oiisjdered to overcome th. computational bottleneck inherent in robot control. First. cad 
aigornhm was ^r. >k.:i nto a m*xk duer.*:n ^v considering which elements were on different levels, which could be run 
.is\ xhror.ousx. ex; which were .omputationallv .merisive. or con id be >.pi *t in?** separate variable types -Fig 4». Then, the 
different a. gonthms were ctMt-tpared to dcvrmme which elements were common Data requirements, computation times, arxj 
output vr each ement were ’abu.ated 























Finally, there re ported dements were cona d ere d in amjmakn with the m fc nnrti oo doer and general system p la cemen t 
requirements to anise at a coherent cocoputef srehi tectn re (Fig. 5). Ptoc eas ot /alg oritom gcrolarity was driven by asaOabfe 
commerc ial p rod a cts and by accuracy requirements. . 

This architecture matches the NASA/NBS standard reference model telerobot control system a r chit e cture (NBS PD far lewis 
one, two, mid part of three. Sesso control and pnramrtrr c al cul a tion take a dvanta ge of parallel proces si n g t eeh alq aes Thsk 
pi inn mg m pwa yacnwnn Mt spur two iucimcocm processes. 



Computer Architecture 


5. ftrfonnance Issues 

Several issues were examined to verily the performance of this parallel co m puti n g system. First, central to the distributed 
control architecture is the correct partitiooment of alogrithms among the parallel processors to properly distribute the comput- 
ing loads. Second, co mimmi ra fio ns bandwidth requirements of algorithm components must be considered to partition the 
algorithms effectively without creating bottlenecks. Third, CPU consumption by algorithm components must be analyzed. 
F inall y, use of operating system and communication system services must also be examined to eliminate unnecessary overhead. 

i Implementation 

The development environment for the digital robot control makes use of Intel microprocessor and microcontroller chip, 
board, and bus technology. The 80286 based System 286/380 is used to run high-level robot path planning code. The system is 
presently used as both a software development and t ar get machine for im ple m e n ting and r unn i ng cootrolsite software. The 
8085-bascd Personal Development System (PDS) is used for developing 8044 microcontroller code, and contains the necessary 
text editor, cross assembler, linkag e editor, and tools for firmware development. An EMV44 eanh i or provides a 

window to the 8044 microcontroller for debug. 

The Distributed Control Module (DCM) package is the set of hierarchical, microcontroller nodes and co m p limen tary com- 
munications software that forms the backbone of the prototype robot control system. It includes the iRCB 44/10 R emote 
Controller Board, the iSBX 344 Multimodule Board, the DCM Controller, and the BITBUS Interconnect. A distributed control 
node may be either m iRCB 44/10 or an iSBX 344 board. The DCM controller is actually an 8044 microcontroller chip with 
built-in SDLC coom rai cntio cs firmware and is part of all boards. The BTTBUS Interconnect is the high-speed serial commum- 
cations link for the network of nodes. 
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The dev el opment configuration was designed specifically to be able to measur e system p erfor ma nce. ll groups six nncrocoo- 
troHers into one distributed network of nodes, tinted together by a high speed serial c ommon i cati oos bos (Fig. 6). The 
co iTmwmic aDoas link inytements a SDLC master slave protocol on chip and perform s all m es s a g e h a n dl in g and error checking 
automatically without help from the CPU. Mastership of the net w ork is daimed by ei th er the Personal Development System 344 
microcontroller node, or the System 286 344 node and is a function of the particular control algorithm currently executing. The 
44/10 microcontroller nodes plug into a common cantfxame which supplies the required power and communications wiring 
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Measuring the system performance is accomplished by attaching an oscilloscope to the pins on one of the dedicated parallel 
I/O ports (Fig. 8). The separate signal lines are toggled *y the development firmware to provide a mechanism for viewing the 
algorithms as they run (Fig. 9). Each bit of the Shit port is dedicated to provide the timing for different components of the 
algorithms running on each processor. 


Measurements were taken in three areas to examine system performance and algorithm partitionment. The three areas 
examined were communications, CPU utilization, and operating system. 
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Serial and parallel message passing techniques differ in the s eq u enc e by which the master node co mmunk ale s with the slave 
nodes. Serial message passing techniques cause the master node to wait for replies after every message sent, which may cause 
other nodes to wait for their tarn if the reply requires some com p utat ion . Parallel message passing techniques aDrav the master 
to have several outstanding m ess a ges, allowing slave nodes mote chance to process in parade! and reduce control 
(Fig. 10). 


The communicatioos bandwidth has significant effect on email timing. With the nodes jum pe r e d for sync hro nou s serial 
co mmun i ca tions on the BTTBUS in t erconnec t, a 2.4 Mbps transmission rate is achieved as comp ar ed with the slower asynchron- 
ous rate (Fig. U). The time savings is readily observed when the master uses serial message passing. 
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Fig. 10 Communi ca tions S ch ama a 
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Fig. 11 Clock Ratss 


CPU utilization measurements verify how effectively the algorithms are partitioned. For example, the slave input nodes 
measured a 60% idle time. This reflects a capacity k> handle more of the computational burden of the system. Compautkm 
measurements for different pans of the algorithm equations enable efficient redistribution of portions of the algorithm id fine 
tune the design. For example, the PD loop timing was approximately 0.8 ms. This measurement helped to redistribute the gain 
computation of the servo loop to a more appropriate node to optimize the cycle time of the loop. 


Operating system calls have a significant effect on algorithm tuning. For example, the preloading of a message for transmis- 
sion by a slave node takes approximately 0.4 ms. Internal RAM buffer space (system pool space) must be allocated for a 
message dynamically, via a system call, if it is to be sent by a task on one node to a task on a different node. Since there is a 
limited amount of buffer space (approx. 4 20-byte buffers) available for a message send, buffer space will be unavailable until 
the buffers associated with previous sends are released after successful transmission. A method of reducing the allocation calls 
at the master node is to use the buffer allocated to hold the previously received reply message for the next order message since 
this buffer is automatically allocated by the system. 


Use of the standard RAC communication services may also impose penalties. Passing a message through the RAC services 
automatically causes task context switching. This is important if the node is on the critical path either in terms of calculations or 
buffer allocations . The time penalties can be avoided by taking direct control of the receive buffers. The applications firmware 
must then provide the communication error handling capability. 


8, Cooctusfoas 


Distributed processing of control algorithms is a feasible solution to improve robot control computatio speed. Measurement 
of system elements is important in order to optimize system CPU usage and avoid bottlenecks. Application programs must be 
careful to avoid unnecessary operating system use. 

This architecture, as implemented, covers the National Bureau of Standards Hierarchical Control Specification levels one, 
two. and part of three ^It is independent of path generation and task dt .cription techniques and runs bilateral force reflection as 
easily as resolved rale. Hardware weight and power consumption are very low and are appropriate for space flight hardware. 
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